Digital phenotyping through multimodal, unobtrusive sensing

The growing adoption of multimodal wearable and mobile devices, such as smartphones and wrist-worn watches has generated an increase in the collection of physiological and behavioural data at scale. This digital phenotyping data enables researchers to make inferences regarding users’ physical and mental health at scale, for the first time. However, translating this data into actionable insights requires computational approaches that turn unlabelled, multimodal time-series sensor data into validated measures that can be interpreted at scale. This thesis describes the derivation of novel computational methods that leverage digital phenotyping data from wearable devices in large-scale populations to infer physical behaviours. These methods combine insights from signal processing, data mining and machine learning alongside domain knowledge in physical activity and sleep epidemiology. First, the inference of sleeping windows in free-living conditions through a heart rate sensing approach is explored. This algorithm is particularly valuable in the absence of ground truth or sleep diaries given its simplicity, adaptability and capacity for personalization. I then explore multistage sleep classification through combined movement and cardiac wearable sensing and machine learning. Further, I demonstrate that postural changes detected through wrist accelerometers can inform habitual behaviours and are valuable complements to traditional, intensity-based physical activity metrics. I then leverage the concomitant responses of heart rate to physical activity that can be captured through multimodal wearable sensors through a self-supervised training task. The resulting embeddings from this task are shown to be useful for the downstream classification of demographic factors, BMI, energy expenditure and cardiorespiratory fitness. Finally, I describe a deep learning model for the adaptive inference of cardiorespiratory fitness (VO2max) using wearable data in free living conditions. I demonstrate the robustness of the model in a large UK population and show the models’ adaptability by evaluating its performance in a subset of the population with repeated measures ~6 years after the original recordings. Together, this work increases the potential of multimodal wearable and mobile sensors for physical activity and behavioural inferences in population studies. In particular, this thesis showcases the potential of using wearable devices to make valuable physical activity, sleep and fitness inferences in large cohort studies. Given the nature of the data collected and the fact that most of this data is currently generated by commercial providers and not research institutes, laying the foundations for responsible data governance and ethical use of these technologies will be critical to building trust and enabling the development of the field of digital phenotyping.I was funded by GlaxoSmithKline and the Engineering and Physical Sciences Research Council. I was also supported by the Alan Turing Institute through their Enrichment Scheme

Digital Phenotyping and Sensitive Health Data: Implications for Data Governance

Mobile and wearable devices, such as smartwatches and fitness trackers, increasingly enable the continuous collection of physiological and behavioural data that permit inferences about users’ physical and mental health. Growing consumer adoption of these technologies has reduced the cost of generating clinically meaningful data. This can help reduce medical research costs and aid large-scale studies. However, the collection, processing, and storage of data comes with significant ethical, security, and data governance considerations. A complex ecosystem is developing, with the need for collaboration among researchers, healthcare providers, and a broad range of entities across public and private sectors, some of which are not traditionally associated with healthcare. This has raised important questions in the literature regarding the role of the individual as a patient, customer, research participant, researcher, and user when consenting to data processing in this ecosystem. Here, we use the emerging concept of “digital phenotyping” to highlight key lessons for data governance which draw on parallels with the history of genomics research, while highlighting areas where digital phenotyping will require novel governance frameworks.I.P.P. work is supported by GlaxoSmithKline and EPSRC through an iCase fellowship (17100053); D.S. work is supported by the Embiricos Trust Scholarship of Jesus College Cambridge, and EPSRC through Grant DTP (EP/N509620/1); J.C. is the recipient of a doctoral scholarship from The Alan Turing Institute and J.M. is supported by the Wellcome Trust

Past, Present, and Future of Multisensory Wearable Technology to Monitor Sleep and Circadian Rhythms.

Movement-based sleep-wake detection devices (i.e., actigraphy devices) were first developed in the early 1970s and have repeatedly been validated against polysomnography, which is considered the "gold-standard" of sleep measurement. Indeed, they have become important tools for objectively inferring sleep in free-living conditions. Standard actigraphy devices are rooted in accelerometry to measure movement and make predictions, via scoring algorithms, as to whether the wearer is in a state of wakefulness or sleep. Two important developments have become incorporated in newer devices. First, additional sensors, including measures of heart rate and heart rate variability and higher resolution movement sensing through triaxial accelerometers, have been introduced to improve upon traditional, movement-based scoring algorithms. Second, these devices have transcended scientific utility and are now being manufactured and distributed to the general public. This review will provide an overview of: (1) the history of actigraphic sleep measurement, (2) the physiological underpinnings of heart rate and heart rate variability measurement in wearables, (3) the refinement and validation of both standard actigraphy and newer, multisensory devices for real-world sleep-wake detection, (4) the practical applications of actigraphy, (5) important limitations of actigraphic measurement, and lastly (6) future directions within the field

The future of sleep health: a data-driven revolution in sleep science and medicine.

In recent years, there has been a significant expansion in the development and use of multi-modal sensors and technologies to monitor physical activity, sleep and circadian rhythms. These developments make accurate sleep monitoring at scale a possibility for the first time. Vast amounts of multi-sensor data are being generated with potential applications ranging from large-scale epidemiological research linking sleep patterns to disease, to wellness applications, including the sleep coaching of individuals with chronic conditions. However, in order to realise the full potential of these technologies for individuals, medicine and research, several significant challenges must be overcome. There are important outstanding questions regarding performance evaluation, as well as data storage, curation, processing, integration, modelling and interpretation. Here, we leverage expertise across neuroscience, clinical medicine, bioengineering, electrical engineering, epidemiology, computer science, mHealth and human-computer interaction to discuss the digitisation of sleep from a inter-disciplinary perspective. We introduce the state-of-the-art in sleep-monitoring technologies, and discuss the opportunities and challenges from data acquisition to the eventual application of insights in clinical and consumer settings. Further, we explore the strengths and limitations of current and emerging sensing methods with a particular focus on novel data-driven technologies, such as Artificial Intelligence

Effectiveness of an mHealth intervention combining a smartphone app and smart band on body composition in an overweight and obese population: Randomized controlled trial (EVIDENT 3 study)

Background: Mobile health (mHealth) is currently among the supporting elements that may contribute to an improvement in health markers by helping people adopt healthier lifestyles. mHealth interventions have been widely reported to achieve greater weight loss than other approaches, but their effect on body composition remains unclear. Objective: This study aimed to assess the short-term (3 months) effectiveness of a mobile app and a smart band for losing weight and changing body composition in sedentary Spanish adults who are overweight or obese. Methods: A randomized controlled, multicenter clinical trial was conducted involving the participation of 440 subjects from primary care centers, with 231 subjects in the intervention group (IG; counselling with smartphone app and smart band) and 209 in the control group (CG; counselling only). Both groups were counselled about healthy diet and physical activity. For the 3-month intervention period, the IG was trained to use a smartphone app that involved self-monitoring and tailored feedback, as well as a smart band that recorded daily physical activity (Mi Band 2, Xiaomi). Body composition was measured using the InBody 230 bioimpedance device (InBody Co., Ltd), and physical activity was measured using the International Physical Activity Questionnaire. Results: The mHealth intervention produced a greater loss of body weight (–1.97 kg, 95% CI –2.39 to –1.54) relative to standard counselling at 3 months (–1.13 kg, 95% CI –1.56 to –0.69). Comparing groups, the IG achieved a weight loss of 0.84 kg more than the CG at 3 months. The IG showed a decrease in body fat mass (BFM; –1.84 kg, 95% CI –2.48 to –1.20), percentage of body fat (PBF; –1.22%, 95% CI –1.82% to 0.62%), and BMI (–0.77 kg/m2, 95% CI –0.96 to 0.57). No significant changes were observed in any of these parameters in men; among women, there was a significant decrease in BMI in the IG compared with the CG. When subjects were grouped according to baseline BMI, the overweight group experienced a change in BFM of –1.18 kg (95% CI –2.30 to –0.06) and BMI of –0.47 kg/m2 (95% CI –0.80 to –0.13), whereas the obese group only experienced a change in BMI of –0.53 kg/m2 (95% CI –0.86 to –0.19). When the data were analyzed according to physical activity, the moderate-vigorous physical activity group showed significant changes in BFM of –1.03 kg (95% CI –1.74 to –0.33), PBF of –0.76% (95% CI –1.32% to –0.20%), and BMI of –0.5 kg/m2 (95% CI –0.83 to –0.19). Conclusions: The results from this multicenter, randomized controlled clinical trial study show that compared with standard counselling alone, adding a self-reported app and a smart band obtained beneficial results in terms of weight loss and a reduction in BFM and PBF in female subjects with a BMI less than 30 kg/m2 and a moderate-vigorous physical activity level. Nevertheless, further studies are needed to ensure that this profile benefits more than others from this intervention and to investigate modifications of this intervention to achieve a global effect

Self-supervised transfer learning of physiological representations from free-living wearable data

Wearable devices such as smartwatches are becoming increasingly popular tools for objectively monitoring physical activity in free-living conditions. To date, research has primarily focused on the purely supervised task of human activity recognition, demonstrating limited success in inferring high-level health outcomes from low-level signals. Here, we present a novel self-supervised representation learning method using activity and heart rate (HR) signals without semantic labels. With a deep neural network, we set HR responses as the supervisory signal for the activity data, leveraging their underlying physiological relationship. In addition, we propose a custom quantile loss function that accounts for the long-tailed HR distribution present in the general population. We evaluate our model in the largest free-living combined-sensing dataset (comprising >280k hours of wrist accelerometer & wearable ECG data). Our contributions are two-fold: i) the pre-training task creates a model that can accurately forecast HR based only on cheap activity sensors, and ii) we leverage the information captured through this task by proposing a simple method to aggregate the learnt latent representations (embeddings) from the window-level to user-level. Notably, we show that the embeddings can generalize in various downstream tasks through transfer learning with linear classifiers, capturing physiologically meaningful, personalized information. For instance, they can be used to predict variables associated with individuals’ health, fitness and demographic characteristics (AUC >70), outperforming unsupervised autoencoders and common bio-markers. Overall, we propose the first multimodal self-supervised method for behavioral and physiological data with implications for large-scale health and lifestyle monitoring.Embiricos Trust Scholarship of Jesus College Cambridge, EPSRC through Grant DTP (EP/N509620/1), GlaxoSmithKline, EPSRC iCase fellowship (17100053

Self-supervised transfer learning of physiological representations from free-living wearable data

Wearable devices such as smartwatches are becoming increasingly popular tools for objectively monitoring physical activity in free-living conditions. To date, research has primarily focused on the purely supervised task of human activity recognition, demonstrating limited success in inferring high-level health outcomes from low-level signals. Here, we present a novel self-supervised representation learning method using activity and heart rate (HR) signals without semantic labels. With a deep neural network, we set HR responses as the supervisory signal for the activity data, leveraging their underlying physiological relationship. In addition, we propose a custom quantile loss function that accounts for the long-tailed HR distribution present in the general population. We evaluate our model in the largest free-living combined-sensing dataset (comprising >280k hours of wrist accelerometer & wearable ECG data). Our contributions are two-fold: i) the pre-training task creates a model that can accurately forecast HR based only on cheap activity sensors, and ii) we leverage the information captured through this task by proposing a simple method to aggregate the learnt latent representations (embeddings) from the window-level to user-level. Notably, we show that the embeddings can generalize in various downstream tasks through transfer learning with linear classifiers, capturing physiologically meaningful, personalized information. For instance, they can be used to predict variables associated with individuals’ health, fitness and demographic characteristics (AUC >70), outperforming unsupervised autoencoders and common bio-markers. Overall, we propose the first multimodal self-supervised method for behavioral and physiological data with implications for large-scale health and lifestyle monitoring.Embiricos Trust Scholarship of Jesus College Cambridge, EPSRC through Grant DTP (EP/N509620/1), GlaxoSmithKline, EPSRC iCase fellowship (17100053